The Guardian University League Table 2014 - Methodology 



1. Summary 

The methodology focuses on subject-level league tables, ranking institutions 
that provide each subject according to their relevant statistics. This ensures 
that all comparisons are as valid as possible - we ask each institution which of 
their students should be counted in which subject so that they will only be 
compared to students taking similar subjects at other universities. 

Eight statistical measures are employed to approximate a university's 
performance in teaching each subject. Measures relate to both input, e.g. 
expenditure by the University on its students, and output, e.g. a graduate's 
probability of finding a graduate-level job. The measures are knitted together 
to get a Guardian score, against which institutions are ranked. 

For those prospective undergraduates who do not know which subject they 
wish to study, but who still want to know where institutions rank in relation to 
one another, the Guardian scores have been averaged for each institution 
across all subjects to generate an institution-level table. 



2. Changes Introduced for 2014 

1 . The methodology employed in the tables has generally remained very constant 
since 2008. There have only been three changes in methodology this year. 

2. Firstly, Integrated Masters courses have been readmitted to the calculation of 
Value Added scores. 

3. Secondly, the interpretation of whether a student undertaking further study is 
representing a positive outcome has shifted from consideration of their mode 
of study (where full time study was always assumed to be a positive outcome 
and part time always negative) to consideration of their level of study (where 
progression to Higher Education or professional qualifications is always 
treated as positive). This change has particularly affected the way in which 
graduates who report that they are engaged in both work and further study are 
treated, and therefore causes a +3% increase in the sector's career prospects 
scores. 

4. The third change relates to the population used in the calculation of average 
entry standards. We have excluded mature learners (those aged 20 or over at 
the point of commencement of studies) from this calculation because of 
concerns over data quality. 

5. To provide more information to the institutions about how their data is used, 
we provide a 'banding boundaries' spreadsheet to show how the point scores 
for Value Added and expenditure per student relate are derived from the 
absolute statistical values. 
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3 . Indicators of Performance 



a. National Student Survey - Teaching 

During the 2012 National Student Survey, final year first degree students were 
asked the extent to which they agreed with four positive statements regarding 
their experience of teaching in their department. The summary of responses to 
all four questions can either be expressed as a percentage who 'definitely agree' 
or 'mostly agree' or be expressed as an average score between 1 and 5 where 5 
relates to students who 'definitely agree' and 1 relates to students who 
'definitely disagree'. The following table gives an example of how a department 
of 30 students might have its data represented in the tables. 







Definitely 


Mostly 


Neither 

Agree nor 
Disagree 


Mostly 


Definitely 


Not 






Agree 


Agree 


Disagree 


Disagree 


Applicable 


The teach 


ingon my course 


5 


4 


3 


2 






1 


Staff are good at explaining things. 


10 


S 


6 


4 


2 





2 


Staff have made the subject 
interesting. 





10 


S 


6 


4 


2 


3 


Staff are enthusiastic about what 
they are teaching. 


2 





10 


S 


6 


4 


4 


The course is intellectually 
stimulating. 


4 


2 





10 


S 


6 


Subtotal 




16 


20 


24 


28 


20 


12 



Satisfaction 


Average 


Rate 


Response 







60% 


3.67 


36% 


2.S6 






S% 


2.33 






25% 


2.33 


31% 


2.85 



b. National Student Survey - Assessment & Feedback 

Students were also asked for their perception of five statements regarding the 
way in which their efforts were assessed and how helpful any feedback was. 





Definitely 


Mostly 


Neither 

Agree nor 
Disagree 


Mostly 


Definitely 


Not 




Agree 


Agree 


Disagree 


Disagree 


Applicable 


Assessm en t a nd feedback 


5 


4 


3 


2 






The criteria used in marking have 
been clear in advance. 


5 


5 


5 


5 


5 


5 


Assessment arrangements and 

6 

marking have been fair. 


10 


8 


6 


4 


2 





^ Feedback on my work has been 
prompt. 


a 


10 


4 


6 





2 


1 have received detailed comments 
on my work. 


14 


1 





14 


1 





Feedback on my work has helped me 
clarify things 1 did not understand. 


1 


14 





1 


14 





Subtotal 


38 


38 


15 


30 


22 





Satisfaction 
Rate 


Average 
Response 




40% 


3.00 


60% 


3.67 


64% 


3.71 


50% 


3.43 


50% 


2.57 


53% 


3.28 



The example data for questions 8 and 9 illustrates how the 'Average Response' 
statistic recognises differences in the distribution of responses whereas the 
'Satisfaction Rate' statistic can be blind to them. This is the reason why Average 
Response is used to rank departments, even though the Satisfaction Rate is 
displayed in the tables. 
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c. National Student Survey - Overall Satisfaction 

Students also answer a single question which encompasses all aspects of their 
courses. 





Definitely 


Mostly 


Neither 

Agree nor 
Disagree 


Mostly 


Definitely 


Not 




Agree 


Agree 


Disagree 


Disagree 


Applicable 


Overall Satisfaction 


Overall, 1 arm satisfied with the 

22 

quality of the course. 


8 


6 


4 


2 





10 



Satisfaction 
Rate 


Average 
Response 




70% 


4.00 



Data relating to the NSS was not released at the JACS level of detail, and results 
had to be weighted in order to approximate Guardian Subject Groups. Level 3 
data carries detail of 107 subjects, but results are suppressed where there are 
fewer than 23 respondents. Where this has happened, we substituted in results 
from level 2, which categorises students into 41 subjects. If any of these have 
fewer than 23 students, our first option is to use level 3 data from the 201 1 NSS, 
otherwise level 2. The last resort is to use the broadest classification of subjects 
- level 1 - to get 201 1 results for the 19 subject groups. 

Caveat: Because the NSS surveys final year students it is subjective and 
dependent upon expectations. Students at a university that generally has a high 
reputation may be more demanding in the quality of teaching they expect. On 
the other hand, students in a department that has been lower in the rankings may 
receive teaching that exceeds their prior expectations and give marks higher than 
would be achieved in a more objective assessment of quality. 

d. Value Added Scores 

Based upon a sophisticated indexing methodology that tracks students from 
enrolment to graduation, qualifications upon entry are compared with the award 
that a student receives at the end of their studies. Each full time student is given 
a probability of achieving a 1st or 2:1, based on the qualifications that they enter 
with. If they manage to earn a good degree then they score points which reflect 
how difficult it was to do so (in fact, they score the reciprocal of the probability 
of getting a 1st or 2:1). Thus an institution that is adept at taking in students with 
low entry qualifications, which are generally more difficult to convert into a 1st 
or 2:1, will score highly in the value-added measure if the number of students 
getting a 1st or 2:1 exceeds expectations. At least 30 students must be in a 
subject for a meaningful Value Added score to be calculated using 201 1/12 data 
alone. If there are more than 15 students in 201 1/12 and the total number across 
2010/1 1 and 201 1/12 reaches 30, then a 2-year average is calculated. 

A variant of the Value Added score is used in the three medical subjects - 
Medicine, Dentistry and Veterinary Science. This is because medical degrees 
are often unclassified. For this reason, unclassified degrees in medical subjects 
are regarded as positive but the scope of the study population is broadened to 
encompass students who failed to complete their degree and who would count 
negatively in the Value Added score. 

e. Student- Staff Ratios 

SSRs compare the number of staff teaching a subject with the number of 
students studying it, to get a ratio where a low SSR is treated positively in the 
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league tables. At least 28 students and 3 staff (both FTE) must be present in an 
SSR calculation using 201 1/12 data alone. Smaller departments that had at least 
7 student and 2 staff FTE in 201 1/12, and at least 30 student FTE in total across 
2010/1 1 and 201 1/12, have a two-year average calculated. Year-on-year 
inconsistency and extreme values at either end of the spectrum cause several 
SSRs to be suppressed or spread over two years. 

Caveat: This measure only includes staff who are contracted to spend a 
significant portion of their time teaching. It excludes those classed as 'research 
only' but includes researchers who also teach, even though at research-intensive 
universities research can take up a significant proportion of their time. It 
therefore follows that the simple ratio of the number of staff to students does not 
accurately reflect teaching intensity and also does not reveal who is performing 
the teaching. Is it the world renowned professor or a graduate teaching assistant? 

f. Expenditure per Student 

The amount of money that an institution spends providing a subject (not 
including the costs of academic staff, since these are already counted in the 
SSR) is divided by the volume of students learning the subject to derive this 
measure. Added to this figure is the amount of money the institution has spent 
on Academic Services - which includes library & computing facilities - over 
the past two years, divided by the total volume of students enrolled at the 
university in those years. Within each department, at least 30 (FTE) students 
have been enrolled in 201 1/12 for the expenditure per student to be calculated. 
Smaller departments must have had 20 FTE in 201 1/12 and at least 30 FTE in 
total across 2010/1 1 and 201 1/12 in order for a two-year average to be 
calculated. Year-on-year inconsistency or extreme values can also cause 
suppression (or spreading) of results. 

g. Entry Scores 

Average Tariffs are determined by taking the total tariff points of 1st year 1st 
degree full time entrants who were aged under 20 at the start of their course, and 
subtracting the tariffs ascribed to Key Skills, Core Skills and to 'SQA 
intermediate 2'. There must be more than 7 students in any meaningful average 
and only students entering year 1 of a course (not a foundation year) with certain 
types of qualification are included. 

Caveat: This measure seeks to approximate the aptitude of fellow students that a 
prospective student can anticipate. However, some institutions run access 
programmes that admit students on the basis that their potential aptitude is not 
represented by their lower tariff scores. Such institutions can expect to see lower 
average tariffs but higher value added scores 

h. Career Prospects 

The employability of graduates is assessed by looking at the proportion of 
graduates who find graduate-level employment, and/or study at an HE or 
Professional level, within 6 months of graduation. Graduates who report that 
they are unable to work are excluded from the study population, which must 
have at least 25 respondents in order to generate results. 
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4. Subject Tables 



Thresholds for Inclusion 

Each Subject table is driven by the eight indicators of performance. An institution can 
only be included in the table if no more than 2 of these indicators are missing, and if 
the institution's relevant department teaches at least 35 full time undergraduates. 
There must also be at least 25 students (FTE) in the relevant cost centre. Under certain 
circumstances an institution can be admitted into a subject table with only 4 indicators 
- if three of the missing indicators relate to the NSS or if the subject is Medicine, 
Dentistry or Veterinary Sciences. 

Standardisation of Scores 

For those institutions that qualify for inclusion in the subject table, each score is 
compared to the average score achieved by the other institutions that qualify, using 
standard deviations to gain a normal distribution of standardised scores (S-scores). 
The standardised score for Student Staff Ratios is negative, to reflect that low ratios 
are regarded as better. We cap certain S-scores - extremely high expenditure and SSR 
figures - at three standard deviations. This is to prevent a valid but extreme value 
from exerting an influence that far exceeds that of all other measures. 



Missing Scores 

Where an indicator of performance is absent, a process introduces substitute S-scores. 



Does institution qualify' 
for inclusion in subject 
table? 



Yes 



Did institution have 
relevant indicator in 
previous year's subject 
table? 



No 



Is indicator correlated 
with other indicators 
wFthi n this subiect? 



No 



No 



Yes 



Yes 




Use S-score from previous 
y ea r. c on strict! ng most 

extreme 10% of results at 
each end of soectrum 



Use average S-Score of institution - 
effectively assuming that HEI would 
have performed as well for this 
indicator as it did for others 



Set absent S-Scoreto zero - effectively 
assuming that HEI would have 
performed similarly to the sector 
average for this indicator 
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Total S-Score and Ranking 

The resulting S-Scores - including those that have been substituted in - are weighted 
according to the values in the following table and added together. 







Weighting in Medicine, 


Indicator 


Usual Weighting 


Dentistry & Veterinary 
Sciences 


NSS - Teaching 


10% 


14% 


NSS - Assessment & Feedback 


10% 


14% 


NSS - Overall Satisfaction 


5% 


7% 


Value Added 


15% 


5% 


Student-Staff Ratio 


15% 


20% 


Expenditure per Student 


15% 


20% 


Entry Scores 


15% 


20% 


Career Prospects 


15% 


0% 



The printed subject table 

The resulting Total S-Scores drive both the subject rankings and the institutional 
table, but are not displayed in the printed subject table. Instead, the Total S-Scores are 
re-scaled so that the institution with the best S-Score receives 100 points and all 
others get a lower (but positive) point score. This statistic appears in the printed 
subject table even though it is not subsequently used in the institutional table. 

In the printed subject table, three of the indicators - entry scores, career prospects and 
Student-Staff Ratios - are displayed in their pure form. The others, however, are not in 
a form that is inherently meaningful to readers. 

Rather than showing the average NSS scores that contribute to an institution's 
ranking, the printed table displays the '% satisfied' statistic because it is easier to 
grasp. Value Added scores are even less inherently meaningful, so the printed table 
displays these as points out of 10, with the following table converting the expenditure 
S-Score into points: 



S-Score Boundaries 



10-point scale 

from to points 

1.8 inf 10 

1 .2 1 .799 9 

0.7 1.199 8 

0.3 0.699 7 

0.299 6 

-0.3 -0.001 5 

-0.7 -0.301 4 

-1.2 -0.701 3 

-1.8 -1.201 2 

-100 -1.801 1 



The same process is used to convert the 
Expenditure per student indicator into 
points. Under certain circumstances it is 
necessary to adjust the boundaries in order 
to ensure that each point score is possible 
to reach - otherwise it would be impossible 
to only score 1 / 10 in a situation where the 
average expenditure per student in the 
sector is less than 1.8 times the standard 
deviation of expenditure, because to do so 
would entail spending a negative amount 
per student. 
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5. Institutional Table 



The Institutional Table ranks institutions according to their performance in the subject 
tables, but considers two other factors when calculating overall performance. Firstly, 
the number of students in a department influences the extent to which that 
department's Total S-score contributes to the institution's overall score and secondly, 
the number of institutions included in the subject table also determines the extent to 
which a department can affect the institutional table. 



The number of full time undergraduates in each subject is expressed as a percentage 
of the total number of full time undergraduates counted in subjects for which the 
institution is included within the subject table. For each subject, the number of 
institutions included within the table is counted and the natural logarithm of this value 
is calculated. The total S-Score for each subject - which can be negative or positive - 
is multiplied by these two values, and the results are summed for all subjects to give 
an Overall S-score for each institution. Institutions are ranked according to this 
Overall S-score, though the value displayed in the printed table is a scaled version of 
this that gives the top university 100 points and all the others a smaller (but positive) 
points tally. 



Each institution has overall versions of each of the indicators displayed next to its 
overall score out of 100, but these are crude institutional averages supplied by HESA 
(or the NSS) that are otherwise disconnected from the tables and give no 
consideration to subject mix. Therefore these institutional averages cannot be used to 
calculate the overall score or ranking position. 



The indicators of performance for value added and for expenditure per student are 
treated slightly differently, because they need to be converted into points out of 10 
before being displayed. Therefore these indicators do read from the subject level 
tables, again using student numbers to create a weighted average. 
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